5a-programming-cover .

Programming exercise 3 – Letter recognition

In this programming exercise, we shall build upon our ANN that we programmed in Exercises 1 & 2, increasing the input complexity significantly such that the ANN is capable of identifying 5x7 pixel images of alphanumeric characters. We will also include the capability for the network to read these inputs from an external source rather than require that the inputs need to be hardcoded into the program itself.

In later exercises, we shall expand this build to:

Make assessments and report upon the network’s performance
Determine when to stop training

In this exercise we shall be building the following network:

ANN for programming exercise 3

Figure 1: ANN for programming exercise 3

Each neuron shall be activated using the Sigmoid Activation Function. Note the significant upsizing of both the input and hidden layers in this network. The input layer is sized so that each pixel in the 5x7 image (35 pixels) can be read independently. The hidden middle layer is sized somewhat arbitrarily to 24. Finally, the number of neurons in the output layer (3) should indicate that, in this exercise, we shall only be seeking to recognise 3 input patterns; the network shall be trained to detect the following input patterns:

input patterns A-C

Figure 2: Input patterns for A-C

Note that for the network to ‘read’ these inputs, a small amount of pre-processing is required. Each neuron in the input layer needs be assigned to a particular pixel, and we shall make those assignments in accordance with this pattern:

Figure 3: Pixel assignment

Using this assignment pattern, the letters A, B and C can be represented as the following string of characters:

	1	2	3	…
A	0	0	1	0	0	0	1	0	1	0	1	0	0	0	1	1	1	1	1	1	1	0	0	0	1	1	0	0	0	1	1	0	0	0	1
B	1	1	1	1	0	1	0	0	0	1	1	0	0	0	1	1	1	1	1	0	1	0	0	0	1	1	0	0	0	1	1	1	1	1	0
C	1	1	1	1	1	1	0	0	0	0	1	0	0	0	0	1	0	0	0	0	1	0	0	0	0	1	0	0	0	0	1	1	1	1	1

In this exercise, we shall write the code for our program to enable it to read these strings of ones and zero in from an external ‘.txt’ file and assign each value to an input neuron. Once these values are read and appropriately assigned, training should commence in a similar fashion to the methods used in previous programming exercises. Note that the training pairs shall be defined as follows:

Training Pairs
Letter	O1	O2	O3
A	1	0	0
B	0	1	0
C	0	0	1

Step 1: Defining the input data (temporarily using internally initialised variables)

In the previous programming exercise, we defined the training pair within the body of the code - in a Training Pair loop that was encapsulated within the Training Cycle loop. Later in this exercise we shall be reading Training Pair data in from an external source but for now we shall initialise some arrays to hold that data so that we can prepare our ANN to process it.

Note that in this step we shall define the known input data (using the variable ‘InputsABC') and associated target output data (using the variable ‘TargetsABC’) prior to the ‘1. Setup ANN Structure’ section of our code. Each of these variables shall be a single array containing all the training pair data.

Initialising the Training Pairs in Code Setup

We shall define the training pairs in two arrays initialised prior to code setup.

Note that at a later step we shall write code that pulls this data from an external ‘.txt’ file.

% Define Training Pairs (Temporary)
InputsABC = ...
[0 0 1 0 0 0 1 0 1 0 1 0 0 0 1 1 1 1 1 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1;
1 1 1 1 0 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0 1 0 0 0 1 1 0 0 0 1 1 1 1 1 0;
1 1 1 1 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 1 1 1];
TargetsABC = ...
    [1 0 0;
    0 1 0;
    0 0 1];

Step 2: Changing the structure of the network

In the previous programming exercises, we defined the size of the network by defining the number of Inputs, Neurons in the Mid Layer and Neurons in the Output Layer (nInputs, nNeurMid and nNeurOut respectively). We then created arrays to store the interconnections between layers, the size of which was based on functions of these variables (“nlayer1w = nInputs*nNeurMid” & “nlayer2w = nNeurMid*nNeurOut”).

We shall make a slight adjustment to this code, such that the input and output layers are automatically sized based on the dimensions of our input data.

We can once again change the structure of our network easily to a 35 x 24 x 3 network by simply changing these variables.

% Structure of ANN
nInputs = size(InputsABC,2);
nNeurMid = 24;
nNeurOut = size(InputsABC,1);

Note: In MATLAB, the command size(arr,2) counts the number of columns in the array ‘arr’ and size(arr,1) counts the number of rows.

Step 3: Modifying our training pair loop to reference externally initialised arrays

Now that our training pair data is initialised external to the body of the code, we need to modify the embedded loop structures of the ANN to reference out to those arrays.

Defining the Training Pair Loop

We shall retain the same basic structure of our embedded training loops but with reference to external data. We therefore want to automate the parameters defining those loops by defining ‘nTPairs’ based on the size of the input data array.

Note: we do not need to change the definition of ‘ErrLog’ as this is already automated.

% Training pairs, cycles and training rate (N)
nTPairs = size(InputsABC,1);
nTCycles = 10;
N = 1;
 
% Array to store output error of each training cycle
ErrLog = zeros (nTCycles, nNeurOut*nTPairs, "double");

Embedding the Training Pair Loop

We shall maintain the embedded Training Pair Loop without modification.

for Tloop = 1:1:nTCycles

     for TPair = 1:1:nTPairs

        % -----------------------------------------------%
        % 2: Define 2 Training Pairs (TPair)
        % -----------------------------------------------%
        % -----------------------------------------------%
        % 3: Forward Pass
        %- ----------------------------------------------%
        % -----------------------------------------------%
        % 4: Backward Pass
        %- ----------------------------------------------%
    end
end

Defining the Training Pairs from an External Source

Now that we have done that, we shall define the training pairs by referencing out to the external arrays.

Note: this means we no longer require the IF statement to initialise the content of ‘KnownIn’ and ‘TargetOut’.

Note: the reference ‘InputsABC (TPair,:)’ is a method for selecting values in the array InputsABC ().

‘TPair’ is the row reference
‘:’ is the column reference and implies ALL values.

‘InputsABC (TPair,:)’ therefore references all values in the row of ‘TPair’.

for TPair = 1:1:nTPairs
   %----------------------------------------------------%
   % 2: Define 2 Training Pairs (TPair)
   %----------------------------------------------------%
    KnownIn = transpose(InputsABC(TPair,:));
    TargetOut = transpose(TargetsABC(,:));

     ...
 
end

Step 4: Plotting the error values

In Programming Exercise 2, we wrote the code to expand the size of the error log automatically based on the parameters of the inputs to the ANN. As such we do not have to make any changes in this exercise:

Figure 4: Example plot of ANN output error following training using 3 letters

Figure 5: Plot of ANN total error following training using 3 letters

Step 5: Reading input data from external file

In our final step of this exercise, we shall read a ‘.txt’ file into MATLAB from an external source to form our Training Pairs. Note this will replace the code we wrote in Step 1 in which we temporarily initialised these variables prior to code setup.

The first thing that we need to do is to create a ‘.txt’ file named [‘ABC.txt’], ensuring that it is within our MATLAB project directory. Into this file, we shall copy/paste the binary strings representing the letters ‘A’, ‘B’ and ‘C’.

Figure 6: Project directory

Figure 7: Screenshot of '.txt' file content

With that completed we can now initialise our training pair arrays, ‘InputsABC’ and ‘TargetsABC’ by reading data from this file:

Placement of Code

Note that the following sections of code need to be located prior to the setup of our ANN structure, i.e., after the introduction to our program:

%{
...
Doug Rattray - doug.rattray@uhi.ac.uk
student ID - Lews Castle College, UHI
Date
ANN – Section 5 - Programming Exercise 3
...
%}

%--------------------------------------------------------%
% 0: Read External Data
%--------------------------------------------------------%
{code to be entered here}

%--------------------------------------------------------%
% 1: Setup ANN Structure
%--------------------------------------------------------%

Prior to reading the file, we shall define how many lines of the ‘.txt’ file that we want to be read, in our case 3: A, B and C.

% Define number of lines to read
nLines = 3;

We can then open the file and define some attributes of the type of data read that we intend to accomplish.

Note this code is reading ('r') the file called 'ABC.txt', specifying it’s format as Base 10 (%d) and floating point (%f) and then sizing an array to store the data of 3 rows (nLines) and however many columns are required for each line (inf).

For more information see the MATLAB reference.

% Read Data from File
fileID = fopen('ABC.txt', 'r');
formatSpec = '%d %f';
sizeA = [nLines inf];

Defining our Training Pairs

We can then populate our InputsABC array variable using the read data.

Note we are also using the MATLAB command eye() to create a square identity matrix based on the number of rows stored in InputsABC. By doing this, we are assigning each row (representing A,B and C) to an output neuron.

% Define Training Pairs
InputsABC = fscanf(fileID,formatSpec,sizeA);
TargetsABC = eye(size(InputsABC,1));

Inputs and targets ABC

Finally, we should close our ‘.txt’ file before proceeding.

% Close file
fclose(fileID);

Once this is done, you may Run your program and check that the ANN is effectively training using input data read from an external source:

Programming Exercise 3